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ABSTRACT 

The purpose of this paper was to identify the characteristics and history of online education, to 
examine research on the effectiveness of online learning, and to note the principles and practices 
indicated for maximizing the effectiveness of online learning. Leading articles and studies on 
online education were identified and examined for this literature review. Historically, online 
education is located within a longer tradition of distance education. The primary distinction 
between campus-based education and distance education is noted, along with distinctions 
between computer-assisted instruction, online learning and hybrid courses. After listing the 
presumed pros and cons of online education, the argument about whether online learning is 
technologically dependent is raised. Prototypical studies of research about online learning are 
examined and used to isolate fundamental problems in carrying out research comparing online 
vs. campus-based education. Major meta-analytic studies are then reviewed. The conclusions of 
these studies are several: there are wide variances in the effectiveness of both campus-based and 
online education; online learning is as good on average as campus-based programs; and the best 
format may be a hybrid or combined version. Standards for delivering online educational 
programs are produced as recommendations, based on integrating standards issued by different 
disciplinary bodies. The paper concludes with a series of issues about online education and 
learning that need to be addressed. (Contains 4 tables in the Appendix.) 
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Over the last 20 years, the growth of online educational programming has been 
spectacular, a conclusion that seems obligatory for articles dealing with online education to state. 
For example, Tallent-Runnels et al. (2006) noted that by the year 2000, virtually 9 out 10 
institutions of higher education (both two- and four- year institutions) offered distance education 
programs. At that time, more than 2.8 million individuals were enrolled in college-level credit 
courses using on-line mechanisms. Grandzol and Grandzol (2006) observed that online 
enrollments for U.S. institutions grew by more than 18% (from 1.98 mi llion to 2.35 million) 
between 2003 and 2004, matching growth rates over the prior three-year period. Business 
disciplines represented the highest levels of growth. 

The purpose of this paper is to describe the theory, the research and the practice of online 
learning. After looking at its historical manifestations, a terminology for this field will be 
offered; then, the potential promise and pitfalls of distance learning will be noted. A major 
question that has driven this field is how online education compares to the more traditional 
classroom-based method. Research on that question will be addressed, including criticisms of 
that research. Next, through several different routes, the prescriptive guidelines for practice will 
be considered. The paper will conclude with questions remaining for research. 

Distance Learning: History, Terms and Consequences 
In spite of its apparent recent popularity, distance education methods can be traced back 
more than a hundred years to the 19th century when improvements in postal services paved the 
way for correspondence courses. Since these early days, distance educational programming can 
be tracked through five different stages or generations (Taylor, 2001); keep in mind that each 
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generation was not eliminated by the next but rather accumulated and persisted with the newly 
emerging versions. The five generations are: 

1. Correspondence Education based on print and postal delivery. (Note: there is probably a 
sixth generation that should be added as really the second stage to distance education; it could be 
called “broadcast education” such as public television in the United States or Britain’s 
University of the Air. Taylor puts mass broadcast radio and television programs in the third 
generation, but this is both a faulty timeline and classification; public television in the United 
States, appearing in the 1950's and 60's, was used to broadcast educational programming a good 
20 years before various teleconferencing modalities were available. See History of public 
broadcasting, n.d.; Shane, 1989.) 

2. A Multi-media model that combined print, audiotape, videotape and computer-disc 
technologies; here, learners purchased the material, then read and completed any activities 
(perhaps with feedback) on their own. An example would be learning a foreign language by 
playing a tape in one’s car's radio player with a workbook provided. 

3. The Tele-learning model of synchronous communications using available technologies, like 
videoconferencing, to provide educational programs. 

4. The Flexible Learning model that used interactive media delivered through the internet for 
online delivery; this included computer-mediated communications. 

5. The current, fifth generation is a so-called Intelligent Flexible Learning model. The main 
offered difference between this and the fourth generation involves "campus portal access to 
institutional processes and resources... [that provides] a customizable e-interface [through which] 
students, staff, and other stakeholders can” interact with the University (Taylor, 2001, p. 10). 
Apparently, this involves allowing 24-7 interactivity by students with essentially all components 
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of the University. It was with the popular diffusion of personal computers in the 1980's, soon 
enough linked by the Internet, that spurred the last stage of distance education, or what we now 
think of as on-line learning programs. 

Terminology and Types 

Given this variegated legacy, it is necessary to use a consistent terminology. Distance 
education (DE) is the most encompassing term that refers to educational practices whereby the 
instructor is physically and geographically separated from students so that instruction must occur 
through some form of media, be it print and mail, radio, television, computers, or otherwise. 
Further, different kinds of delivery vehicles can be identified, such as correspondence courses 
that distributed printed texts through the mails; television; and now Web-based approaches 
(Shachar and Neumann, 2003). Within the framework of Web-based education, several 
additional distinctions can be identified (Tallent- Runnels et ah, 2006). On-line courses are those 
that are delivered completely through the Internet, while hybrid or blended courses combine 
online with traditional classroom instruction. When Web-based courses use specific, 
transaction-mediating software, that is e-learning. (Presumably, courses could be delivered 
through the Internet using e-mail but without benefit of some kind of educational, online 
software, such as Blackboard.) It is likely that, today, most online courses use some sort of 
educational transaction software, so for this paper. E-learning courses will be called online. 

Distance education can also be distinguished from classroom-based, traditional, face-to- 
face (f2f) education (terms used interchangeably here) and from computer-assisted instruction 
(CAI; also computer based training or CBT). In CAI, instructional modules are programs 
embedded in a computer. These programs involve predetermined tutorial instruction segments 
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as well as drill-and-practice exercises; more enriched versions may include graded evaluations of 

performance along with additional feedback or guidance for problem-solving. There is 

interactivity between the instructional and testing elements of the program and the student. 

These instructional experiences are typically done by the individual, rather than in a group 

format. In short, CAI instruction is particularly suited for tutoring, drill-and-practice, and 

testing. A 1991 meta-analysis of 254 studies of computer-assisted instructional programs was 

conducted by Kulik and Kulik. They found that CAI students had significantly more learning 

(0.3 standard deviations higher) than students in conventional classrooms. 

We conclude, therefore, that the typical student in an average CBI (computer-based 
instruction) class would perform at the 62nd percentile on an achievement examination, 
whereas the typical student in the conventionally taught class would perform at the 50th 
percentile on the same examination. Put another way, the average student from the CBI 
class will outperform 62 percent of the students from the conventional class, (p. 80) 

Interestingly enough, the stronger effect sizes occurred with programs whose length of time was 
no longer than four weeks; that is, learning effectiveness decreases with the increasing length of 
time of CAI usage in a class. 

One other distinction that is common in classifying programs is the extent to which they 
are synchronous (instructors and students are communicating online simultaneously or at the 
same time) or asynchronous (where there is no immediate communication taking place). 

On-Line Learning: Arguments Pro and Con 

During the period of time over which online education has evolved, numerous claims 
about the benefits or advantages of online education have been offered. Likewise, there been 
numerous disadvantages noted (Grandzol and Grandzol, 2006). For example, the following 
advantages of online education have been suggested: 
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• Improving the technical literacy of students 

• Minimizing projected shortages in instructors 

• Alleviating overcrowding and/or reduced investment in college physical plants 

• Increasing enrollment and profits 

• Creating a more friendly learning environment 

• Allowing students to work at their own pace 

• Extending geographic reach and presence of an institution 

• Improving graduation rates 

• Reducing costs associated with commuting and other work demands. 

These anecdotally noted advantages have been contrasted to these disadvantages: 

• Poor quality of instruction 

• Training costs for faculty 

• Evoking faculty resistance to change 

• Eack of student-teacher interactions 

• Employer skepticism 

• Increased faculty workloads 

• The inappropriateness of the medium for teaching certain types of course content 

• Problems in technology and administration 

• Eoss of scholarly control. 

DE: Technology Neutral or Technology Dependent? 

Given the many conveniences and economies that DE courses promise, interest in the 



relative effectiveness of distance education has provoked both researcher and critic. Clark 
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(1994) took an early position in this debate, contending that technological media, including 
computers and the Internet, are relatively inconsequential in their effects on learning, particularly 
compared to more powerful factors like instructional methods and individual learner differences. 
That is, there are really two kinds of technologies involved: instructional technologies use 
social-psychological research in organizing content into instructional programs in order to 
produce learning, while delivery technologies focus on providing efficient and timely access to 
those instructional packages. Delivery technologies affect the cost and access to instructional 
material, but do not have an independent effect on the quality of instructional programs. 
Moreover, he criticized research comparing DE and face-to-face instruction on several counts. 
Most studies, for example, do not have adequate experimental controls to rule out competing 
explanations of outcomes. Further, he argued that there may be unique institutional practices for 
different media, and studies typically do not isolate those unique instructional factors. In 
summary, for Clark, technology is neutral and not determinative of learning outcomes. 

This argument butts against the question, not of learning efficacy per se, but rather the 
situational or domain suitability of online learning in the context of certain types of occupations. 
Specifically, the American Association of College of Nurses (2000) wonders how well online 
learning prepares students for the social and behavioral skills needed in a field noted for its 
people-intensive work: “Can students learn to relate well in a multi-disciplinary environment 
when their dominant educational experiences have been technology-based, essentially isolated 
from classmates and teachers except for telecommunications?” (p. 3). This question has not 
been adequately researched. A fundamental issue is the extent to which fields that involve 
people-intensive tasks actually require direct, on-going, interpersonal contact with others 
(faculty and classmates). Alternatively, if the nuances of reading and responding to others can 
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be clearly identified (my guess is that such a listing of the social and behavioral skills associated 
with people-intensive work has not adequately been performed), can those skills be taught using 
online mechanisms? The default assumption - what should be a hypothesis - is that they cannot. 

Traditional Vs. Online Education: How Do They Compare? 

While studies comparing traditional versus other forms of distance education go back a 
number of years, there is been a consistent and growing body of research since the early 1980's 
that has addressed the fundamental questions about whether on-line education is as good as 
traditional, face-to-face instruction. The study by Dellana, Collins and West (2000) is 
illustrative of the typical research paradigm used. Two sections of the same quantitatively- 
oriented management science course were compared. This course, typically taken by junior 
level students, was taught by the same instructor, and covered the same content with 
approximately equal class size and instructional approach. The traditional version was taught in 
a lecture format with Power Point visuals, while the on-line course presented the same material 
online to students who could not or wished to avoid attending the classroom lecture. The on-line 
portion of the course included lecture material with Power Point format along with a discussion 
room and e-mail. There was a total of 221 participating students (70 in the traditional course 
format); 7% of students in the traditional course dropped out vs. 11% in the Online version. All 
exams for both sections were given in an on-campus classroom. There was no statistically 
significant difference between the traditional and the online versions on average scores. 

Several items of note about this study illustrate issues throughout this stream of research. 
First, there are several internal threats to the validity of this study. (Threats would be any flaws 
in the research design that could explain the obtained result apart from the stated independent 
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variable of the course delivery vehicle.) For example, students were not randomly assigned to 
the different conditions but rather self-selected the instructional format they would use. Thus, 
one cannot rule out preexisting differences as accounting for some of the differences in the 
scores; that is, people would select the option that is best for them. (In general, this issue raises 
the possibility that there can be an aptitude-treatment interaction at work, the practical 
implication of which is that students may need to be either prescreened for participating in online 
classes and/or students given a choice as to the type of venue they use.) Another potential threat 
might involve differences in instructor experience doing in-class v. online programs. That is, an 
instructor who is extremely versed in classroom teaching could presumably offer a different 
quality learning experience than if he had to do the same course online and for which his 
experience was limited (Anstine and Skidmore, 2005). In the Dellana, Collins and West (2000) 
study, the instructor had been involved for many years in providing distance education programs, 
so experience level could have been ruled out here; even so, the potential threat remains. 

Second, the attempt was to make the on-line class as much of a duplicate of the regular 
traditional course as possible. This is an important issue which will be addressed again. For 
now, the question is whether this is a fair basis for comparison. A draft horse could be fairly 
compared to other draft horses but would be unfairly compared to race horses. In other words, is 
an online, computer and web-based learning process really the same as a classroom process 
(obviously, minus the physical presence of an instructor) or is the online process a different 
"animal"? Hypothetically, how is the quality of each course delivery technology rated? Is a low 
quality lecture being compared to a high-quality on-line course? Alternatively, simply 
duplicating classroom instruction onto an online form does not insure that the online form is the 
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best possible structure for online delivery. Indeed, in this context, duplication may produce 
misleading outcomes. 

Third, in terms of the reporting of these studies, the description of the specific techniques 
used in both the online and the traditional classroom methods leaves much to be desired. The 
classroom method was defined as a lecture format but did this mean a virtual non-stop 
presentation by the instructor? How much student interaction — with each other, with the 
instructor — was there? Were there ever class exercises or activities? Likewise, what happened 
in the online sessions was only broadly explained. At issue here is difficulty in interpreting what 
the actual treatments (class experiences) really were. In short, how different were the treatments 
and in what significant ways? Without a much more detailed explanation, it is hard to evaluate 
what is being compared. 

Fourth, test grades were used as the basis for comparison. While this is imminently 
understandable, issues remain in several ways. For example, using Bloom's (1956) hierarchy of 
learning outcomes, it is possible for tests to sample different levels of complexity of knowledge 
and cognitive processing, from simple recall of isolated facts to much more nuanced evaluation 
and analysis. The latter pole would ask about the extent to which critical thinking is being 
advanced. Tests likely include items from across the spectrum of learning outcomes, but an 
issue is whether there are any differences in effects of either form of instruction in terms of 
testing levels. There is also the issue of subject domain. In the Dellana, Collins and West (2000) 
study, the subject matter was technical in its nature: there were clear procedures in the form of 
formula applied to problems in which all needed information is provided in the text and for 
which there is a single right answer. But what of other domains in which there is extensive 
human interaction, for which judgment (not technical equation solving) is essential, when 
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information must be discovered, and when there are a variety of possible solutions of similar 
costs and benefits? 

Finally, the "no significant difference" finding indicated that the online instruction was as 
effective as in-class instruction. As we will soon see, this basic conclusion of "no significant 
differences" between the types of delivery vehicles is a consistent and pervasive baseline 
conclusion. This conclusion, though, has been interpreted in several ways, all of which are 
justifiable. Proponents of online education are quick to seize on the findings, pointing to the 
equivalency of the techniques. Critics, perhaps relying upon implicit promises of the purported 
superiority of on-line programs, point to the failure of on-line programs to deliver better results. 
Thus, one's ultimate reaction to these findings will depend upon one's initial expectations. 

Online v. Traditional, On-campus education: Research summaries 

The number of studies comparing traditional versus online courses is surprisingly large to 
an observer first visiting this field. Indeed, the number of studies is so substantial that a number 
of research summaries, including several meta-analytic reports, have been produced. It is to 
those reports that we turn. 

An early entrant into this context was Russell (1999). As reported in Zhao, Lei, Yan, Lai 
and Tan (2005), Russell cherry picked more than 350 studies, from 1928-1999, that supported 
the finding of no significant differences in learning outcomes for DE sources. While this source 
supported the early claim that there was no significant difference between DE and classroom 
based instruction, his method neutralizes any claims that come from this analysis. 

Shachar and Neumann (2003) identified 86 studies (representing more than 15,000 
students) between the years of 1990-2002 that had no severe methodological flaws, included a 
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control or comparison group, and had sufficient quantitative data to calculate an effect size (the 
difference between the means of the two sets of data divided by the pooled standard deviation of 
the sample). One-third of the studies found negative results (that is, traditional instruction 
outperformed online classes), while two-thirds found positive effects. Of that group, 35% found 
a small effect, while the balance was either medium or large sized (19%). After combining all 
analyses, a combined or pooled single effect sizes was then calculated. This overall effect was a 
significant, medium effect size favoring online instruction (d-i- =.366): "The final academic 
performance grades of students enrolled in distance education programs are higher than those 
enrolled in traditional face-to-face programs" (page 13). The authors, however, did not provide 
adequate detail about the studies that they included, specifically in terms of student grade levels 
or course content in the studies used. Neither did they provide, as is customary, an identification 
of the specific studies that they used. Even though the results are suggestive, questions remain 
about the sources of data that were used. 

At least three meta-analyses, though, do deserve close attention and respect because of 
their rigorous methodology and analysis. Each will be reported in turn. 

The most inclusive study was conducted by Bernard and his associates (2004). They did 
an exhaustive search to identify studies of Distance Education and identified 232 studies from 
1985 (an arguable milestone when PC and web-based educational venues reached a level of 
popularity) through 2002. DE was defined in terms of the relatively permanent separation, both 
in time and space, of learner and instructor during planned learning events that were conducted 
under the aegis of a recognized educational organization and that included two-way media for 
dialogue and interaction between instructor and students. Eurther, the studies had to include an 
empirical comparison of the DE format with a face-to-face (f2f) classroom format. The level of 
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learners (from kindergartners to adults) had to be identified, and the studies had to be just for 
courses, not entire programs. Three different kinds of dependent variables were noted: 
achievement outcomes (n = 321), attitudes (n = 262), and retention (n = 105). Studies were 
coded in terms of a variety of features, including course design formats, media use, subject 
matter, and so on. As is common in such procedures, effect sizes for each study were weighted 
to create an overall estimate of average treatment effect. 

Findings were presented in terms of the three dependent variable measures. First, in 
terms of achievement outcomes, the data represented almost 55,000 students. In general, there 
was a very small yet significant difference in achievement scores favoring DE over classroom- 
based education; the difference was just slightly better than zero. At the same time, there was 
large variability in outcome scores for both formats. In other words, the data indicate that some 
distance education applications are much better than classroom instruction, and some are much 
worse. Asynchronous distance education programs, however, were significantly better while 
synchronous programs were significantly worse. Further analysis indicates that the factors that 
were significantly responsible for the differences were attributable to the methodology employed 
in the research (such as type of publication, type of measure, treatment duration, instructor 
equivalence, class size equivalence, and so on) and pedagogical features of the instruction (such 
as whether a systematic instructional design procedure was used, whether there was f2f contact 
involved, the nature and amount of contact between students and/or with the teacher, or the use 
of problem-based learning). Media or technology factors did not predict achievement. In short, 
methodological issues — the structure of the research — accounted for a substantial portion of the 
variance. While there were no differences between undergraduates students, graduate school 
applications yielded modest, significant results in favor of DE. Eace-to-face instruction was 
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found to be better for topics such as math, science, and engineering, while computing and 
military /business topics seem to be slightly better in distance education. (Findings about people- 
intensive fields of practice, like nursing, counseling, education or Human Resource 
Development, were not reported.) 

The research on attitudes were grouped into four categories: attitudes about technology, 
the subject matter, the instructor, and the course. (Unfortunately, in this report, results were 
aggregated and not reported for each category.) There were 154 attitude outcome measures 
available. In general, DE was significantly associated with negative attitudes. A significant 
negative difference held for synchronous but not for asynchronous DE. Again, there was a large 
variance in attitude scores. And again, methodological factors accounted for the lion's share of 
variance, more so, though, for synchronous courses. Retention rates were significantly lower for 
online courses. Once more, there were differences between synchronous and asynchronous 
courses, with the latter showing significantly more dropout rates of students. 

They drew the following overall conclusions. Eirst, there are substantial weaknesses in 
the research methodologies used that substantially compromise the quality of conclusions that 
can be used for policy making. Eurther, there was relatively little guidance that can be extracted 
from this corpus of research. Second, there is wide variability in outcome measures for both DE 
and f2f formats. The overall opinion that DE is better or worse than f2f courses is simply not 
supportable. That is, both DE and f2f work very well, or they can work very poorly. They were 
not able to isolate what specific factors might be responsible for those differences. 

A second study by Sitzman, Kraiger, Stewart and Wisher (2006) found 96 research 
reports from 1996 to 2005, representing more than 19,000 students in 168 courses, both 
educational and work-based; 67% of the reports involved undergraduate students. They coded a 
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number of features of the courses included, providing a finer detailed analysis of moderator 
variables. The primary dependent variables were declarative knowledge (of facts and principles) 
and procedural knowledge and skill. In line with the prevailing conclusions of the field, "web- 
based instruction" was significantly but weakly more effective than traditional classroom 
instruction in teaching declarative knowledge. There was no evidence of differences between 
web-based and classroom-based instruction on procedural knowledge. Students and trainees 
were equally satisfied with both delivery mechanisms. 

On top of these now-to-be-expected outcomes, several new findings emerged. First, the 
courses that combined a Web-based and classroom-based instruction (so-called "hybrid" 
courses) outperformed either method separately on both declarative and procedural knowledge. 
Indeed, hybrid courses were more effective for teaching procedural knowledge. Other important 
moderating variables included the amount of learner control, practice and feedback included in 
the course. Specifically, when Web-based courses allowed higher levels of control, practice 
opportunities and feedback to students, the effect of web-based instruction was strong compared 
to classroom instruction. Interestingly enough, human interaction had no effect in distinguishing 
web-based from classroom-based instruction. Students and trainees tended to react more 
favorably to stand-alone classroom instruction than to hybrid programs. Finally, in the few 
studies (n=ll) in which learners were randomly assigned to instructional conditions, classroom- 
based instruction was more effective. They observed that there may easily be a person-treatment 
interaction effect; that is, when people self-select the mode of instruction, they are likely to pick 
the method in which they feel most comfortable and competent. 

The third meta-analysis by Zhao et al. (2006) found 51 applicable studies, yielding 99 
effect sizes for almost 12,000 students. Like the Sitzman et al. (2006) study, they were able to 
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code an extensive number of factors describing the type of characteristics of courses. Again, in 
general agreement with field, on-line courses were slightly more effective than face-to-face 
courses, however, the difference in this study was not significant. Likewise, there was extensive 
and larger variance in effect sizes for both forms. 

They found the following significant outcomes. First, studies published after 1998 were 
significantly more likely to report distance education being more effective. They also found that 
the kind of outcome measure used explained some of the differences; specifically, student self- 
assessments of learning tend to slightly favor face-to-face education; when other common 
metrics, including grades or student attitudes are measured, they significantly favor distance 
education. (That is, DE students think they are not learning as much when in fact they are). 

Like Sitzman et ah, they also found that the combination of distance and face-to-face instruction 
(hybrid courses) seemed to be most effective. In general, they found three factors that favored 
distance education: instructor involvement, media involvement (hybrid), and the type of 
interaction used (both synchronous and asynchronous interaction was best). Unlike Sitzman et 
ah, though, instructor involvement was a significant moderator; indeed, it was the most 
significant moderator of outcomes. "When instructor involvement is low, the outcome of 
distance education is not as positive as those of face-to-face education; when instructor 
involvement increases, distance education programs yield more positive outcomes than face-to- 
face education" (page 33). 

Summary of the meta-analytic research 

The meta-analytic reviews of distance education agree on two major points: first, 
distance education is as effective (if not slightly more so) than traditional classroom-based, face- 




Distance, On-line Education 18 



to-face instruction on many measures of academic performance; and second, there is a large 
range of variance in both forms of educational delivery. Alternatively, there is substantial 
opportunity to improve both online and classroom-based instruction; even further, there is a lack 
of quality consistency in education, regardless of the delivery mechanisms used. Such a 
conclusion supports the original position of Clark (1994) that technology was instructionally 
neutral, or in other words, that it is pedagogical methods — not technology systems — that are 
determinative in producing learning outcomes. 

This research has also identified a number of factors that seem to differentiate courses in 
terms of their effectiveness. There are some methodological artifacts, such as the date of the 
study and type of outcome measure used, that produce some of the variance in results. This 
research is also cognizant of multiple threats to study validity, such as differences in teacher skill 
or lack of equivalency between course structures. There does seem to be some agreement that 
individual preferences and abilities are important entry conditions; specifically, when students 
can select the type of delivery mechanism they prefer, better learning results. In other words, 
preexisting differences among learners are more than a threat to validity; they are important 
exogenous variables to consider in assessing instructional effectiveness. A number of other 
pedagogical and instructional practices have been identified as having some role. Often, the 
instructional practices are not delivery-specific but rather refer to good pedagogical techniques 
being used, regardless of the delivery method. In this context, consider the importance of 
practice for learning skills or techniques: computer-based training modules may actually be a 
superior way to provide practice; such a method can be used both in classrooms and through on 
line delivery. This research also has noted several factors about which the results are equivocal, 
such as the relative value of synchronous vs. asynchronous communications, the importance of 
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the instructor presence, and so on. Finally, there does seem to be agreement that hybrid or mixed 
media courses may actually be the most effective. 

Limitations to and criticisms of these comparative studies 

Research comparing online vs. in-class educational courses has its critics and criticisms. 
A general set of criticisms against such research has been offered by Lockee, Burton and Cross 
(1999). For example, they note these persistent weaknesses in the basic research protocols: lack 
of adequate theory, inadequate sample sizes, weak implementation of the online treatment 
condition, and measurement flaws. Moreover, particularly for experimental comparative studies, 
the standard is that all else is the same except for the one manipulated independent variable. To 
the extent that all other possible differences have not been removed, questions of interpretation 
remain. For example, if the instructor(s) used in such comparative studies has (have) different 
degrees of experience teaching online vs. in-class, there can be subtle yet nonetheless noticeable 
effects on the delivery of the course and the quality of the outcomes (Anstine and Skidmore, 
2005). Since pedagogical practices seem to make a big difference in outcomes, this factor could 
be an important confounding variable in comparative analyses. 

Another critical issue involves the reliability and validity of the measurement instruments 
being used; in this context, the instruments are usually learning achievement tests. These tests 
are often made by the teachers but without assessments of reliability or validity. When testing 
instruments of unknown psychometric quality are used for grading purposes (as is typically the 
case in the studies), there is a potential instrumentation threat to validity. Critics devote special 
attention to the use of the null hypothesis of no significant differences as a justifying conclusion 
of the research. Such a finding does not prove that on-line education is as good as classroom 
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education. Indeed, accepting the null hypothesis does not refute the tested or alternative 
hypothesis of differences. 

There are two persistent threats to the validity of studies comparing DE v. f2f classes. 
One has already been alluded to: the differential retention rates of online and f2f courses. The 
threat is, of course, subject mortality. As reported by the Bernard and associates (2004) meta- 
analysis, online courses have significantly higher drop-out rates than f2f classes. A reasonable 
inference (one that would need verification, though) is that the students who drop out of an 
online course are probably the ones who are doing poorly, not the ones who feel confident in 
passing it successfully. Thus, the students left in the Online class at the time of the final 
examination will likely be at an upwardly biased or skewed level of learning. In other words, it 
reasonable expectation is that without the higher dropout levels, the average achievement scores 
of students taking on-line courses could easily be lower. Whether this would be enough to 
produce a significantly lower outcome is not known. One other check on this point would be to 
look at studies where there are minimal if any differences in dropout rates between online and 
face-to-face courses in order to determine if there are differences in achievement outcomes. 

The other threat to validity comes from pre-existing differences. Often, it is simply 
impossible in higher educational settings to randomly place students in on-line or face-to-face 
courses. That is, students typically self-select the course format they will use. This self- 
selection factor can function to produce effects or outcomes independent of whatever 
instructional format is being used. As Lockee, Burton and Cross (1999) put it: 

participants in higher education courses are typically non-traditional learners who cannot 
attend class at the originating institution, hence their enrollment in distance programs. 
Not only are the students different demographic ally, but they also possess of the 
characteristics which vary from traditional college attendees, such as prior knowledge 
and experience and level of motivation, (p. 5). 
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This issue can be seen in a study by Shulman and Sims (1999). They studied 
achievement outcomes of students in either face-to-face or online versions of the same courses. 
Five different courses, representing several different disciplines, were used. The same 
instructors taught each respective pair of courses, and students self-selected the course format. 
Both pre- and post-test measures were collected. As would be expected, there were no 
significant differences between the online and the face-to-face scores at the end of the semester. 
However, there were significant differences in the pre-test scores; specifically, students enrolled 
in the on-line courses scored significantly higher than did the students in the face-to-face version 
(average online pretest scores was 40.70, while the average pretest scores for the in class 
students was 27.64). In other words, more learning occurred in the in-class formats than in the 
online versions. By using pretest measures, an entirely different understanding of the 
effectiveness of online, distance education emerges. Indeed, one is left wanting a new 
metaanalysis that compares change scores, using pre and post measures, between DE and face- 
to-face courses. Such a focus might also change the nature of the question from: Is there any 
difference in learning achievement scores between the two formats, to: Which format produces 
more learning? 



Distance Education: Guidelines and Standards 
If online education can be as good as in-class education, what should be done to ensure 
the highest consistent level of quality learning in on-line courses? As noted earlier, the very 
wide distribution of effect sizes for both online as well as in-class programs indicates a strong 
need for identifying and implementing policies and procedures that will narrow that gap and 
produce consistently high educational and learning outcomes. Various attempts have been made 
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to address or establish guidelines for structuring online education. Several of those attempts will 
be noted below, followed by an integrative listing. 

One approach to determining recommendations for effective online instruction can be 
through actual evaluations of courses, as can be seen in the evaluation of four on-line courses 
conducted by Gramm and his colleagues (2000). They used an evaluative framework based 
upon a model of good practices in undergraduate education (Chickering and Gamson, 1987). 
That is, according to this model, there are seven principles for effective instruction: student- 
faculty contact, encouraging cooperation among students, promoting active learning, prompt 
feedback, emphasizing time on task, communicating high expectations, and respecting diverse 
talents and ways learning. From this evaluation, a number of recommended practices were 
inferred. (Those recommendations will be integrated into the framework reviewed shortly.) 

A second source of guidelines comes from prescriptive reports and assessments. For 
example, Kazan and Schnorr (1999), noting the widespread opportunities for interaction and 
feedback between students and their teachers in online programs, observed that such 
opportunities are seldom used. Rather, educators often use the Web simply to distribute static 
documents. Instead, feedback and assessment procedures can be implemented to monitor 
progress in learning as well as evaluate teaching approaches. They recommend using HTML 
Forms (a special editing program available in such products as Microsoft Frontpage or Adobe 
PageMemo) that can be used to create data collection and reporting forms. "HTML forms have 
designated fields in which the user enters information that is sent back to the course instructor 
via the web. In most cases, students fill out the feedback form and press the 'Submit' button to 
automatically forward student-entered data to faculty as an e-mail message" (page 31). 
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Procedures enable interaction and feedback, engaging students in both participating and reacting 
to comments. 

A third source of practice guidelines can be found in standards-setting policies adopted 
by various accrediting agencies. The process of online education cannot be divorced from the 
accrediting issues that may be involved. "In 2001, the eight U.S. regional accrediting 
commissions, in an attempt to hold on-line programs to high standards, collectively created a 
best practices statement to assist institutions and facilitating on-line programs" (Grandzol and 
Grandzol, 2006, p 4). The resulting document (Best Practices, n.d.) used five separate 
components of distance education programs (see below) as a framework for making 
recommendations. Other attempts at doing the same have been offered by the accrediting agency 
of business schools, the Association to Advance Collegiate Schools of Business (AACSB) for 
standards for online education for business programs; and the American Association of Colleges 
of Nursing (AACN) for online nurse educational programs. These various recommendations 
have been consolidated below using the Best Practices, five-component model as the organizing 
framework. (When practices are drawn from the other sources, that source is indicated by one of 
these acronyms: CSB= AACSB; CN=AACN; TWB= Graham et al.. Teaching in a web- based 
distance learning environment). 

I. Institutional context and commitment . Online programs create a new and potentially very 
different extension of the historical role and functioning of the institution. A critical issue then is 
the extent to which this extension both fits with and is supported by the institution. Particulars: 

• The Institution’s budgets and policies should reflect the commitment to students who are 



learning on-line. CSB 
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• The institution insures adequate technical and physical facilities, including appropriate 
staffing and technical assistance, to support electronic on-line programming; this requires 
substantial financial investment in technology, infrastructure and faculty development. 
CSB, AACN. 

• The internal organizational structure for operating the on-line programs maintains 
appropriate academic oversight that ensures the integrity of student work and faculty 
instruction. The institution ensures that DE learning outcomes are of comparable quality 
to on-campus programs. CSB 

• The institution evaluates courses and programs on their learning outcomes, not in terms 
of their modes of delivery. 

• Institutions should undertake distance education for reasons other than achieving a 
financial windfall. CSB 

II. Curriculum and instruction . As already suggested in the research, pedagogical practices are 
particularly important in determining the quality of online education. Particulars: 

• The institution assures that each program of study produces a level of learning 
appropriate to the rigor and depth of the degree or certificate being awarded. 

• The focus of educational efforts should shift from the traditional concern for teaching to a 
broader emphasis on producing learning effectiveness. CSB 

• Academically qualified professionals participate fully in the decisions about the program 
curriculum and program oversight. Include a variety of stakeholders in planning. CSB 

• The institution provides a way for students to access all courses necessary to complete 
the program or will notify them of requirements not available electronically. 
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• Appropriate interaction between the instructor and students, and among students is 
reflected in the design of the program and its courses; appropriate technical facilities and 
support services enable the interaction. E-mail communication policies should be clearly 
communicated, such as hours for availability, response time, and so on; students who are 
falling behind or not regularly participating in online discussions should be contacted by 
the instructor. CSB, TWB 

• Student interactions should be encouraged through both well-designed group projects as 
well as the well-designed discussions; students can participate in face-to-face meetings at 
the start of the semester to build a sense of community. The assignments should include 
meaningful interaction. Likewise, there should be a means to evaluate individual 
participation and contribution to group work [to counter social loafing] . TWB 

• Students should present their work to the rest of the class, and class members should be 
encouraged to give feedback. TWB 

• Students should be given a structure of regular work assignments and schedules for the 
entire semester the will provide a disciplined framework for staying with material. TWB 

"Educators should design learning experiences to take advantage of various modalities that best 
fit with learning objectives and with student learning styles" (AACSB, 2007, page 10). 

in. Eaculty support. Particularly for faculty used to traditional instruction, online instruction 
involves new pedagogical approaches and new skills (CSB). Providing appropriate support by 
the institution is necessary. Particulars: 
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• The institution and faculty have addressed the issues of workload, compensation, 
ownership of intellectual property, and the implications of online teaching for the faculty 
member’s professional evaluation. CSB ^ 

• The institution provides ongoing technical, design, and production support for faculty. 

• There is a climate that encourages and rewards innovation and risk-taking. AACN 

• The institution provides appropriate training in technology, course design and 
management. 

• Faculties should understand and support the change from a teaching-centered to a 
learning-centered environment. They should be both creative and willing to learn new 
teaching methods. AACN, CSB 

• Faculty members should be encouraged to learn from each other in their experiences with 
online teaching. This can be done formally or informally. Awards for teaching 
innovations may be useful, too. TWB 

According to Graham et al. (2000), " instructors are generally motivated to do an excellent job of 
teaching in an online environment but are not always up to date with what strategies will be most 
successful in the online teaching environment" (p. 19). 

IV. Student support . Appropriate institutional practices provide appropriate support for 
students. Particulars: 

• The institution has a commitment (administrative, financial, and technical) to continue a 
program for a period of time sufficient to enable all admitted students to complete the 



degree or certificate in the normal time frame. 
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• The full range of student support services should generally be the planned and made 
ready prior to the implementation of distance education programs. Prior to admitting 
students to a program, the institution informs prospective students about required 
technology and technical competence required. Students also are told of library and other 
learning services available. CSB 

• The design of the program attempts to develop a sense of learning community through 
such activities as encouraging study groups, providing student directories, limitations to 
campus events, and similar activities. 

V. Evaluation and assessment . Both the assessment of student achievement and the evaluation 
of the overall program take on added importance in online situations. This is especially an issue 
in asynchronous programs, when seat time or time on task cannot be adequately assessed or 
controlled. Particulars: 

• There is documented assessment of student achievement in each course; at the 
completion of the program, student performance is compared to intended learning and 
outcomes. 

• Testing situations are conducted in such a way as to assure the integrity of student work 
by validating student identity. 

• There are documented procedures to assure the security of personal information in 
assessments. 

• Program effectiveness should include measures of student achievement against intended 
learning outcomes, student retention rates, and satisfaction levels for both students and 
faculty. 
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• The institution conducts a program of continual self-evaluation of its on-line programs. 

Tallent-Runnels and associates (2006) completed a more traditional literature review of 

online studies. They found 76 studies, both quantitative and qualitative in nature, from which 

they extracted a list of lessons about online learning. Lessons were organized in four major 

categories: course environment, learner outcomes, learner characteristics, and institutional and 

administrative factors. For purposes here, the lessons can be identified as either pros, cons, 

issues or cautions, and guidelines. This list is summarized in Appendix I. In general, they 

concluded that few definitive guidelines can be derived from the existing research. Nonetheless, 

some conclusions seem warranted. First, some students — but not necessarily all — prefer to 

move at their own pace and schedule through a course. This is likely to be an individual 

difference factor, though, reflecting such traits as personality, self-management or study habits, 

and other factors. Second, the quality of learning that occurs in an online environment is 

significantly affected by the quality of the instruction provided. 

Not surprisingly, students in well-designed and well-implemented on-line courses learn 
significantly more, and more effectively, than those in on-line courses where teaching 
and learning activities were not carefully planned and where the delivery and 
accessibility are impeded by technology problems. This finding challenges online 
instructors to design their courses in accordance with sound educational theories. 

(P.116). 

Applications and Implications 

Surprisingly, a number of recommendations that have emerged tend to focus on factors 
and conditions outside the immediate online instructional experience. In particular, the 
positioning of online courses within the broader institutional framework in terms of its suitability 
and support enjoyed widespread agreement. As important in this framework is the provision of 
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adequate technical and administrative support. Another important set of recommendations 
involved instructor/faculty preparation. But this goes beyond simple faculty training to include 
policies and decisions on compensation, workload, and P&T matters. Another point of emphasis 
has been on adequate communications to (including promotional information) and preparation of 
students. In a sense, all these factors refer to critical enabling conditions that increase the 
chances of effective instructional design and delivery through online environment. 

Another critical finding in this regard is the apparent neutrality of technology on 
conventional measures of learning achievement. That is, using relatively standard testing 
procedures, there seems to be little difference in overall effectiveness between classroom-based 
and on-line courses. Instead, the immediately critical factor responsible for the variances and 
quality of education is the pedagogical practices employed. To date, I have not seen an 
assessment of online pedagogical practices that have compared top and bottom performing 
classes in terms of pedagogical procedures used. Instead, it appears that the recommendations 
for pedagogical practices are based more on the experience and informed conjecture. It seems 
likely that well-established research-based guidelines for the best pedagogical practices are still 
waiting. 

That being said, certain types of pedagogical practices seem to enjoy relatively 
widespread support. These conclude following: focused yet regular communications and 
interactions between the instructor and students, as well as among students themselves. There 
are some technical details about best methods for communicating information. Adams and 
Morgan (2007) note that e-learning instruction has shown itself effective in teaching “technical” 
but not “soft” skills. 

. . .this approach can be effective for delivering expert knowledge and technical training, 

especially when there are right or wrong answers which learners need to understand. It is 
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not as effective for education, training and development in soft-skills areas where the 
answers to predetermined questions depend on the detailed nature of the problems that 
have to be addressed.... (p. 165) 

They argue that technical skills training (what they call “first generation” e-learning approaches) 
are instructor-controlled, predetermined linear learning programs where learning focuses on 
memorizing rules of practice, repetitive drill and frequent passing of embedded testing. A 
“second generation” approach that puts the learner in control of a discovery-based process using 
reflection and self-assessment for integrating knowledge would be more suitable for developing 
soft skills. A second- generation e-learning system is question -based and encourages the learner 

to explore material in ways that are unique and specific to the person. They report data on a 

2 

program they developed implementing this approach (NewMindsets) that was effective. 

Unanswered Questions — or Unknown Answers? 

As a working paper, this document represents a first attempt to summarize the research 
literature regarding distance education and on-line learning. An important and apparently robust 
finding is that, in general, online education is as good as more traditional, campus-based courses. 
This conclusion exists in the larger context of extremely wide variation in the quality of both 
delivery approaches. Beyond that conclusion, though, finding more solid recommendations for 
policy and practice is difficult. In short, it appears that there are a number of important yet still 
unanswered questions. Its not clear, though, whether the research has not caught up to the 
questions, or whether I have simply not caught up to the research. 

Regardless, it is useful to identify some of the critical issues that still need to be 
addressed, in my view, in order to round out our understanding of the practice of online 
education and learning. Those questions are presented below in no particular order. 
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1. In the meta-analysis by Bernard and his associates (2004), differences were noted between 
synchronous and asynchronous courses. Why would there be such differences based upon the 
timing of dialogue and communication circumstances? 

2. Computer assisted instructional modules have a stronger effect than distance education in 
general. Clearly, computer assisted instructional modules can be delivered as effectively online 
as in a campus-based computer lab. How can CAI modules be best included in distance 
educational courses? 

3. What are the best pedagogical practices for online learning? In turn, what do those practices 
mean for the role of the teacher, and what kind of skills must an online teacher possess? Further, 
how does the job performance characteristics of online teaching profile against traditional 
education? For example, the traditional time spent lecturing in a classroom would typically no 
longer apply; so how do online teachers use their time? 

4. In general, what is the range and variety of on-line help pedagogical practices? For example, 
experiential learning techniques (see Silberman, 2007) offer a number of methods for engaging 
students in activities as the foundation for learning. To what extent can those practices be 
modified or adapted to online instruction? What kinds of experiential methods would not be 
adaptable? And are there unique pedagogical techniques for online learning? 

5. How important are individual difference factors in explaining or supporting learning 
achievement and on-line programs? Some research supports the effect of individual differences. 
What differences are important, and what are the implications of individual differences for 
online instruction? When should those differences be assessed, with what instruments, and how 



should that information be used? 
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6. In terms of specialized applications, one could argue that graduate-level education can be 
distinguished from undergraduate education not only in terms of its depth but also in its focus. 
Specifically, graduate education often is oriented towards professional practice, whether it be in 
medicine, technology, business, psychology, nursing, or other fields. The issue is what is the 
proper structure for educating for professional practice? At least four different areas of emphasis 
can be identified in this regard. First, there needs to be knowledge of basic structure and 
process. Medical doctors need to learn both anatomy and physiology. Second, there needs to be 
drill and practice in techniques. Business students must learn how to conduct financial analysis 
or how to do a marketing plan, skills presumably improved through multiple practice 
assignments. Third, there is the need for diagnostic talents, which involve actual data collection 
in field conditions. Engineers must learn how to evaluate sub-surface soil and rock conditions in 
putting up a building, for example. These diagnostic procedures may often involve applying the 
techniques already learned but in the context of actual operating conditions. Finally, practicing 
professionals must learn to be able to operate using the principles of evidence-based practice. 

Did the patient recover? Did organizational revenue increase? Did the building stand as planned? 
In this framework, the design of the educational curriculum should be based on the educational 
methods and techniques which will best produce the learning outcomes needed. In other words, 
pedagogy should trump delivery. To what extent can online education appropriately and 
effectively implement each of the purported stages of professional education? 

Online learning has become the growth product in the educational realm. Given all its 
apparent attractiveness, particularly the convenience it offers to students and its potential 
revenue enhancing capabilities at minimal investment cost, on-line learning acts as a powerful 
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magnet in curricular design. The evidence for online education is not dismissive but is rather 
supportive. Yet the potential for weakening educational quality through online learning seems as 
great as its potential for improving it. In order to move forward, more secure footing is needed; 
identifying how to make distance education as consistently effective as possible is no doubt the 
essential next step. 



Endnotes 

1. Milheim (2001) argues that institutions need to make policy decisions about the following 
compensation issues: Do you pay for course development and/or delivery of the course? How 
should compensation be restructured: in regular salary compensation, as a percentage of 
royalties, or as course release time? In addition, he notes that teaching a distance education 
course is potentially risky to faculty in several ways. First, the heightened workload can 
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compromise or reduce time devoted to scholarship and thereby threaten promotion and tenure 
opportunities; further, since students in online courses tend to rate faculty teaching lower, that 
can impact current merit review and reward outcomes. Such issues are particularly problematic 
for new faculty members. (The increase in teaching time and commitment comes from several 
sources in on-line programs: increased preparation time for various instructional materials; 
higher needs for interaction with students online; and potential travel time and/or meetings with 
students at remote sites.) 
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Appendix I. Lessons from the Literature. Derived from Tallent-Runnels, et al. (2006) and Grandzol and Grandzol (2006). 



Table 1. Course Environment that includes: 1) 

2) 

3) 

4) 

5) 



Classroom culture 
Structural Assistance 
Success factors 
Interaction systems 
Evaluation system 



Pros/Advantages 


Cons/Disadvantages 


Issues and Cautions 


Guidelines 


1) CLASSROOM CULTURE. 
In asynchronous formats, 
students have more time to 
think about their responses 
which should improve depth of 
processing, 

Anonymity can be an 
advantage 


OE courses do not embody 
“conversational language” 

OE formats tolerate 
unchallenged ideas 


Instructors may ignore student 
affective states in online 
discussions which can be 
harmful 


Establish a community of learners by 
setting up study groups, modeling 
effective communications, or having 
initial and/or periodic f-2-f meetings. 

Eearner-focused designs are critical 


2) STRUCTURAE 
ASSISTANCE. The self- 
pacing nature of OE learning is 
appreciated by students. 




[Students need real-time 
interactions with instructors; 
in OE, this is probably 1-1 but 
extends demands on 
instructor’s time. 

Observation: 

OE is convenient to students 
but increases instructor’s 
workload. Question: whose 
time is more important?] 


Motion/video presentations enhance 
online instruction above simply 
providing slides 

The pedagogically most important 
information should be presented via 
video 

Instructors should provide 
“instructional scaffolds” [advance 
organizers?] as guiding questions 
help students progress thru the 
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OE courses require higher 
degrees of student self- 
management and direction 


material 

Students need time interacting with 
instructors to help them learn 
material better. Students value email 
communications with the instructor. 


3) SUCCESS FACTORS 


Some number of students 
(41%) do not like learning 
from the WWW. Some 
number prefer listening 
rather than reading. 


Eight factors affect student 
success: access to tools, 
experience with technology, 
learning preferences, study 
habits, goals, purposes, 
lifestyles, and personal traits. 

Students spend more time on 
course material and work in 
classroom rather than OE 
formats 


Instrument for assessing student 
learning preferences: Online 
Eearning Environments Survey 

One cannot simply duplicate an in- 
class design onto an online format 

One must design online courses to 
meet learner expectations and 
cognitive styles 

Students value and appreciate 
discussion. 


4) INTERACTION 
SYSTEMS. The actual ratio of 
time-on-learning task / total 
classroom time should be 
higher in OE formats [but this 
comes from studies of pre- 
higher education schooling; 
that finding may not hold for 
higher education] 


The availability of other 
media, both online, 
radio/TV, entertainment 
systems and so on present 
the opportunity for 
distractions while OE 
sessions are in progress 
[how do you know if 
student’s are actually on- 
task and paying attention?] 


One study found that students 
in OE settings process 
information at relatively low 
and superficial levels while 
rarely being engaged in 
knowledge negotiation, 
refinement and construction. ^ 


Guidance or structures for 
participation should be geared at the 
deeper levels of processing. 

Instructors should provide prompt 
feedback and encourage social 
interaction and collaborative 
learning 

Synchronous discussions are better 
for social interaction; asynchronous 
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Online discussions allow 
everyone to participate (no 
competition for airtime) 






discussions, for task processing. 

The instructor should avoid trying to 
respond to every student comment. 
Delegate duties in discussions: give 
various students duties to manage 
and/or summarize discussions. Ask 
other students to answer questions. 


5) EVALUATION: no specific 
lessons 









1. “Students learn only when their current view of knowledge is challenged, reformed, and synthesized through their interaction with 
others.” P. 100. 



Table 2. Learner Outcomes: 1) cognitive and 2 ) affective. 



Pros/Advantages 


Cons/Disadvantages 


Issues and Cautions 


Guidelines 


1) 


Students separated from 
instructors may have a greater 
opportunity /incentive to turn in 
purely plagiarized work than 
when an instructor is present 




Instructional material that is 
organized in a very structured, 
linear fashion is better for 
students that are low self- 
regulators 

An aptitude-interaction model 
will allow better integration of 
personality and instructional 
factors 


2) Online learners most value 
the ability to set their own pace 
and scheduling. In-class 
students most valued 
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interactions with the instructor 
and other students. 









Table 3. Learner Characteristics 



Pros/Advantages 


Cons/Disadvantages 


Issues and Cautions 


Guidelines 








For any kind of course, its 
important to recognize the 
student’s goals, needs and 
motives 



Table 4. Institutional and Administrative Aspects: 

1) Institutional policies 

2) Institutional support 



Pros/Advantages 


Cons/Disadvantages 


Issues and Cautions 


Guidelines 


1) 






There should be institutional 
policies for OL education 


2) 




Faculty believe they should be 
paid for developing the course. 
They believe that OL courses 
take more time 


OL instructors want training 
that is used on demand, has 
follow-thru and support 
resources. They wanted the 
Dept chair to be visible. 

This should include support 
and assistance for both course 
development and delivery 







